Method and apparatus for visualizing a medical instrument under ultrasound guidance

ABSTRACT

Systems and methods of facilitating the viewing of interventional instruments (e.g., needles, catheters, guidewires, vascular filters, biopsy probes) are disclosed herein. In one embodiment, a portable, handheld or cart-based ultrasound imaging machine is connected to an external laser light source that transmits laser light to a tip of an interventional instrument via one or more optical fibers. The laser light is absorbed at the distal tip of the instrument and generates a photoacoustic signal. The ultrasound imaging machine creates an image by combining data from the received photoacoustic signals with ultrasound echo data to show both tissue in a region of interest and the tip of the interventional instrument.

RELATED APPLICATION

The present application claims the benefit of, and priority to, U.S.Provisional Application No. 62/430,298 filed Dec. 5, 2016, which isherein incorporated by reference in its entirety.

TECHNICAL FIELD

The following disclosure is generally directed to systems for detectingone or more interventional instruments in a subject. In particular, oneor more aspects of the disclosed technology are directed toward systemsfor producing a photoacoustic signal at an end portion of a needle andfor producing an ultrasound image in which the end portion of the needlecan be seen.

BACKGROUND

Many medical procedures require the accurate placement of an instrumentin the body. For example, during a nerve block procedure, a physicianand their assistant attempt to place a needle for delivering anestheticnear a particular nerve. Because the tip of the needle is in the bodyand can bend as it is being advanced and the exact location of the nerveis unknown, different techniques and tools are available to help thephysician determine if the needle tip is in the correct location. Forexample, nerve stimulators can help confirm the location of a needle tipin the proximity of a nerve by applying a small amount of electricity tothe patient's body to stimulate the nearby nerve. If the correct muscleassociated with the targeted nerve responds to the electricity, theoperator knows that he/she has reached the proximity of the target nerveto be blocked, and can then inject a drug.

Conventional ultrasound needle guidance technology can also be used todetermine a needle and nerve location. One method, for example, involvesenhancing and/or modifying ultrasound imaging parameters to emphasizethe shaft of a needle that is in the same plane as the ultrasound image.Another method involves the use of an echogenic needle havinglaser-etched patterns on the needle shaft that enhance specularreflectivity when the needle is placed in the subject at a steep angle.Some predictive methods infer the location of a needle and its tip usingmagnetic and/or optical sensors attached to an ultrasound transducerand/or a needle. These predictive methods can be cumbersome, however,adding bulk and cost to an ultrasound transducer and typically requiresubstantial training. Furthermore, since the location of the needle tipis inferred, a bent needle can lead to inaccuracies in the predictedneedle tip location.

Another approach that is being developed is to use photoacoustics tolocate a needle tip. With this method, one or more laser light pulsesare transmitted to an optical absorber at the distal tip of a needlethat cause the absorber rapidly heat and expand on a microscopic scalecompared to the surrounding tissue. The expansion creates ultrasonicvibrations that can be detected with an ultrasound transducer. Whilesome ultrasound machines are being designed with the capability tointerface with a laser source that delivers such laser pulses and detectthe corresponding echo signals, there is a need to be able to use suchtechniques with older or less sophisticated ultrasound machines, or withmachines that are not specifically designed to support a photoacousticimaging mode. The disclosed technology simplifies the complexity andreduces the requirement for large a memory and a powerful processor thatare typically required in conventional photoacoustic imaging.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an interventional instrument imaging systemconfigured in accordance with an embodiment of the disclosed technology.

FIG. 1B illustrates one embodiment of a needle for use with theinterventional instrument imaging system shown in FIG. 1A.

FIG. 1C is a functional block diagram of an ultrasound imaging machineof FIG. 1A during a photoacoustic imaging procedure in accordance withan embodiment of the disclosed technology.

FIG. 1D is a timing diagram showing an example of a transmission andreceiving sequence of interleaved ultrasound and photoacoustic linesignals in accordance with an embodiment of the disclosed technology.

FIG. 1E is a timing diagram of a transmit and receive sequence forultrasound lines received for a B-mode imaging frame followed by anumber of photoacoustic lines received for a photoacoustic imaging framein accordance with an embodiment of the disclosed technology.

FIG. 1F shows interleaved blocks of ultrasound lines received for aB-mode imaging frame and blocks of photoacoustic lines received for aphotoacoustic imaging frame in accordance with an embodiment of thedisclosed technology.

FIG. 2A is a functional block diagram of an ultrasound imaging machinedesigned to produce an image of a tip of an interventional instrumentusing photoacoustic signals in accordance with some embodiments of thedisclosed technology.

FIG. 2B shows a signal processing path in an ultrasound imaging machineto produce an image of tissue and a photoacoustic image of aninterventional instrument in accordance with some embodiments of thedisclosed technology.

FIG. 2C shows how the timing of received photoacoustic signals in anultrasound imaging machine can be adjusted to compensate for a delay ina laser firing in accordance with some embodiments of the disclosedtechnology.

FIG. 2D illustrates how B-mode and needle tip visualization (NTV) dataare combined to create an image in accordance with an embodiment of thedisclosed technology.

FIG. 3 is a functional block diagram of an external laser light sourcein accordance with an embodiment of the disclosed technology.

FIG. 4 is a representative screenshot of an ultrasound image withincluding an image of a needle tip in accordance with an embodiment ofthe disclosed technology.

FIG. 5 is a flow diagram of a process of forming an ultrasound image inaccordance with an embodiment of the disclosed technology.

DETAILED DESCRIPTION

The disclosed technology relates to systems and methods of facilitatinginterventional procedures such as, for example, the administration ofanesthesia, MSK/sports medicine for joint/tendon/muscle injection, fineneedle aspiration, amniocentesis, thoracentesis, pericardiocentesis,vascular access and biopsy. In one embodiment, a system includes anultrasound imaging machine and an external laser light source thatcommunicates with the ultrasound imaging machine to deliver light pulsesto a tip of an interventional instrument (e.g., a needle, a catheter, abiopsy instrument, guidewire, vascular filter or the like). Theultrasound imaging machine is configured to transmit ultrasound energyinto a region of interest of a subject (e.g., a human or an animal) andreceive corresponding ultrasound echoes from the subject and theinstrument. The laser light source generates one or more laser lightpulses that are transmitted to a tip or a distal portion of theinterventional instrument via one or more optical fibers. The distal endof the fiber is covered with a light absorbing material such as a blackepoxy. The external laser light pulses are absorbed by such material inan area adjacent the distal tip of the interventional instrument, whichcauses photoacoustic signals to be produced.

In one embodiment, the light absorbing material is positioned at the tipof the fibers to absorb the laser light pulses and emit correspondingphotoacoustic signals that the ultrasound imaging machine can detect.The ultrasound imaging machine receives the photoacoustic signals andproduces real-time line data for use in creating an ultrasound image ofthe tip of the interventional instrument based on the detectedphotoacoustic signals. In some embodiments, the ultrasound imagingmachine is configured to produce an ultrasound image of theinterventional instrument using a color map. The machine can process thesignals to form a colorized ultrasound image that is superimposed on orcombined with, a normal gray-scale tissue image. As a result, the usercan visualize a colored set of pixels in a combined ultrasound image toshow the instrument tip in real time. The machine can be configured toreceive a user command to turn this feature on/off and control theintensity or brightness of the colored image via gain or laser outputpower manipulation within the allowed regulatory requirements. Themachine can also be configured to receive user input regarding a desiredcolor for the displayed image of the instrument tip. In someembodiments, a signal to automatically turn on the imaging mode is sentfrom the external laser light source when an interventional instrumentis connected to reduce or simplify the user interaction. These and otherembodiments of the disclosed technology are expected to provide morenatural hand-eye coordination, more precise placement of theinterventional instrument tip, reduced procedure times and/or enhancedinstrument tip visualization compared to conventional instrumentvisualization techniques. This is especially helpful for steep angleinsertions where it has been difficult to image a needle usingtraditional ultrasound imaging techniques. For out-of-plane insertions,the disclosed technology can tell the user if a needle tip has reachedthe imaging plane.

FIG. 1A is a partially schematic view of an interventional instrumentimaging system 100 including an ultrasound imaging machine 110 coupledto an ultrasound transducer 120, an external laser light source 140 andan interventional instrument 130 (e.g., a needle). FIG. 1B is aschematic view of a shaft portion 134 of the interventional instrument130. Referring to FIGS. 1A and 1B, the ultrasound transducer 120 isconfigured to transmit ultrasound energy into a region of interest 102of a subject 101 and receive corresponding ultrasound echoes from theregion of interest 102. A cable 122 carries electronic signals producedin response to the received ultrasound echoes from the transducer 120 tothe ultrasound imaging machine 110. The ultrasound imaging machine 110processes the electronic signals and generates one or more ultrasoundimages 113 that are displayed on a user interface 112 of a display 111.An input interface 115 receives user input and instructions via one ormore user input controls (e.g., one or more buttons, keys, knobs,switches, sliders, trackballs and/or touch-sensitive surfaces).

Referring to FIGS. 1A-1C together, the external laser light source 140(including a laser diode, miniaturized YAG, Alexandrite or other type oflaser light source) communicates with a port such as an EKG port of theultrasound imaging machine 110 via a connection cable 146 (e.g., a USBcable, I2C cable, ECG cable, an HDMI cable or a custom designed cable).In some embodiments, however, a wireless connection (e.g., Bluetooth,802.11 etc.) can be used for communication between the laser lightsource 140 and the ultrasound machine 110. One or more optical fibers142 extend from the laser light source 140 to a tip 135 (FIGS. 1B and1C) of the shaft 134. In some embodiments, the one or more opticalfibers 142 extend through an interior portion of the shaft 134. In otherembodiments, however, the one or more optical fibers extend on anexterior surface of the shaft 134. Yet in another embodiment two or morechannels of the needle could be used to house the fiber specifically.The one or more optical fibers 142 can be attached to the interior orthe exterior surface of the shaft 134 with an epoxy or another adhesiveto allow room for fluid flow inside the shaft of the interventionalinstrument 130. In some embodiments, a double or multi-lumen instrumentseparates the one or more optical fibers 142 from a fluid channel. Insome embodiments, the exterior of the shaft 134 is free of markers,while in other embodiments, an outer surface of the shaft includes oneor more markers that are used to indicate the depth to which shaft isinserted into a body.

As will be explained in further detail below, the external laser lightsource 140 includes a system interface for power and communication, amemory for storing a device ID and program instructions, an opticalassembly including a mechanical connector for engaging a fiber opticconnector, a light shutter, a light ring and one or more LEDs thatilluminate when the laser is activated.

The laser light source 140 is configured to produce one or more fixed orvariable wavelength laser light pulses in the range of visible orinvisible IR light (300 nm to 1500 nm) as an example, that aretransmitted to the tip 135 via the one or more optical fibers 142. Theduration of the laser pulses is selected so that the photoacousticsignals created at the tip of the instrument are in the receivebandwidth of the ultrasound transducer 120.

In some embodiments, a light absorbing medium 148 (FIG. 1B) ispositioned at the tip 135 and covers the ends of the one or more fibers.The absorbing medium absorbs the one or more laser light pulsestransmitted from the laser light source 140 and generates photoacousticsignals 143 (FIG. 1C). As shown in FIG. 1C, the ultrasound transducer120 transmits ultrasound pulses 118 into a region of interest anddetects ultrasound echoes 119 as well as the photoacoustic signals 143that are created in response to the laser pulses. As explained infurther detail below, the ultrasound imaging machine 110 forms one ormore ultrasound images of tissue and the interventional instrument 130in the region of interest 102 using the detected ultrasound echoes 119and the photoacoustic signals 143. As those of ordinary skill in the artwill appreciate, because the photoacoustic signals 143 originate onlyfrom the needle tip, the source of the photoacoustic signals 143 is thetip location. In one embodiment, the ultrasound imaging machine 110creates data for two images (e.g. from the returned ultrasound echoesand the received photoacoustic signals respectively), and combines thedata for both images to create a combined image in which the colored tipof the instrument is shown to the user among the normal gray scaletissue image. In some embodiments, the pixels representing the tip ofthe instrument are colorized differently from a traditional gray scaleultrasound tissue image to increase the contrast or awareness of the tiplocation. Note the photoacoustic image is an ultrasound real time imagecreated in response to photoacoustic signals that originate directlyfrom the location of the needle tip, not a graphical indicator that isderived from some other inputs. Therefore, it is not prone to thebending of the needle shaft and is applicable for any angle of insertionincluding out-of-plane methods.

In some embodiments, the light absorbing medium 148 at the distal end ofthe instrument is an epoxy that absorbs the laser light pulses and emitscorresponding photoacoustic signals 143 (FIG. 1C). As those of ordinaryskill in the art will appreciate, the absorption of the individual laserlight pulses increases the temperature of the light absorbing medium148. The resulting temperature increase causes the light absorbingmedium 148 to expand and produce the photoacoustic signals 143 that canbe detected by the transducer 120. Because the photoacoustic signals 143are only emitted from the tip 135, the ultrasound imaging machine canproduce an image of the tip 135 directly without using another method toinfer the location of the shaft 134 and/or the tip 135.

In some embodiments, the light absorbing medium 148 may comprise, forexample, an epoxy, a polymer, a plastic and/or another suitable materialthat can stick to the fiber(s) and absorb the laser light to generatethe photoacoustic signals. One benefit of using a light absorbingmaterial is prevention and/or reduction of light leakage into the tissuethat itself could generate photoacoustic signals, which could causeambiguity of the needle tip location. In other embodiments, the ends ofthe one or more optical fibers are silvered or otherwise coated with ametal or other material that absorbs the laser pulses. In still otherembodiments, the optical fibers are angled, cut or lensed so that thelaser light pulses are directed onto the distal end of theinterventional instrument or into the tissue. The material that absorbsthe pulses then generates the photoacoustic signals in a manner that issimilar to that of the black epoxy.

Line-Based Imaging

In the illustrated embodiment of FIGS. 1C and 1D, the ultrasound imagingmachine 110 (e.g., a portable ultrasound machine) is configured tosupport line triggered laser firings and to use conventional beamformingpaths to form photoacoustic image lines that are interleaved with linescreated for B-mode imaging. In one embodiment, the ultrasound imagingmachine 110 is configured to interleave the ultrasound and laser pulsefirings. The machine transmits the ultrasound pulses 118 and receivesthe corresponding ultrasound echoes 119 to form a line of a B-modeultrasound image frame. It then switches to the photoacoustic imagingmode by halting or suspending the ultrasound machine's transmissionelectronics or by causing the transmit electronics to produce ultrasoundsignals with little or no energy and sending one or more trigger signalsto the external laser light source 140 to cause the production of one ormore of the laser pulses. Photoacoustic signals are then created inresponse to the one or more laser pulse firings.

In the embodiment shown in FIG. 1D, ultrasound lines (US) for a B-modeimage frame are alternately produced with photoacoustic lines (PA) for aphotoacoustic image frame. Alternately, as shown in FIG. 1E, an entireset of N ultrasound signals (US1-USN) can be created for a B-mode imageframe followed by a set of laser firings to produce a set ofphotoacoustic lines (PA1-PAN). Alternating blocks of ultrasound signalsand laser firings can also be obtained as shown in FIG. 1F In someembodiments, one or more laser pulses are transmitted for each line usedin making a photoacoustic imaging frame.

FIG. 2A illustrates further detail of one embodiment an ultrasoundimaging machine that is programmed to produce images of a needle tipusing the photoacoustic signals. The ultrasound system includes one ormore processors 117 that are programmed to execute needle tipvisualization (NTV) software when an operator wants to visualize theposition of a needle using photoacoustics. After connecting the externallaser light source 140 to a port (EKG, USB or other port) 152 on theultrasound machine, the laser light source 140 communicates a device IDto the processor 117 to inform the machine that the laser light source140 is connected. In one embodiment, power for the external laser lightsource 140 is provided from the ultrasound machine to the laser lightsource 140 through the port 152. However, the laser light source 140could be powered by a battery or from another power source (externalpower supply) if desired.

Conventional photoacoustic imaging is done mostly by obtaining radiofrequency element data and using the data to form a photoacoustic image.The beamforming is done not using an existing delay-and-sum hardwarebased beamformer but a fast CPU or GPU are programmed to dosoftware-based beamforming. Since this approach is slow compared to ahardware-based approach, a fast beamforming method called FFTbeamforming is used. However, it only applies to a linear transducer. Toaccommodate both normal imaging and photoacoustic imaging, either thenormal imaging has to be moved to the FFT-based software beamformer oradditional processing power has to be provided to support both methods.In some embodiments described, this obstacle is alleviated by notrequiring a software-based beamformer but instead utilizing an existinghardware-based beamformer for photoacoustic imaging.

As indicated above, some ultrasound imaging machines do not havespecialized hardware support for photoacoustic imaging (also referred asneedle tip visualization mode—NTV). For these ultrasound machines, theoperating software is modified so that the machine can produce data fortwo images that are blended together where the data for one of theimages is created from the received photoacoustic signals. In accordancewith one embodiment of the disclosed technology, when operating in theNTV mode, the processor 117 executes NTV software instructions 119 thatcause the processor 117 (or the transmit electronics 114) to generate atrigger signal (labelled Tr) when the laser light source 140 is toproduce a laser light pulse. In some embodiments, the processor 117 alsoinstructs the transmit electronics 114 to reduce the amplitude or theduration of a transmit pulse at the time when the laser light pulse isproduced by the external laser light source so that little or noultrasound energy is produced by the ultrasound transducer 120. Uponreceipt of the trigger pulse, the laser light source starts the sequenceof producing one or more laser pulses, which cause photoacoustic signalsto be generated near the needle tip. Receive electronics 116 in theultrasound system 110 are then enabled to detect the photoacousticsignals.

A transmit/receive switch 156 is used to protect the sensitive receiveelectronics 116 from the large voltages produced by the transmitelectronics 114 and other transients. After one or more pulses have beentransmitted by the ultrasound transducer 120 for the acquisition of aline for a B-mode imaging frame, the position of the transmit/receiveswitch 156 is changed by the processor 117 so that the receiveelectronics 116 begin detecting the return echo signals from a desireddistance away from the transducer (e.g. the skin line). During aphotoacoustic line acquisition, the receive electronics are controlledto begin receiving signals from the same positions away from thetransducer as the lines in the ultrasound frame for correct spatialregistration of the PA frame with the normal B-mode frame. In someembodiments, the position of the transmit/receive switch 156 can remainclosed after a B-mode line is received. Keeping the switch closed avoidselectrical noise associated with opening and closing the switch 156 thatcauses artifacts.

Echo signals created in response to the B-mode firings are two-waybeamformed and signal processed and may be stored in a B-mode imageframe memory 160 until all the line data required to produce a frame areobtained. Similarly, one-way beamformed and signal processedphotoacoustic signals may be stored in a second image frame memory 162(labelled photoacoustic PA memory). Once all the line data for bothframes are obtained, the processor 117 combines data from each frame toproduce a composite image in which the tissue in the region of interestand the position of the tip of the instrument can be seen.

As indicated above, because the ultrasound imaging system 110 is notspecially designed to perform photoacoustic imaging, the system uses theexisting receive electronics 116 to process the photoacoustic signals.In one embodiment, the system is programmed to operate as if ittransmits ultrasound from the imaging transducer but the transmitelectronics are controlled to transmit pulses with minimal or no energyby reducing the amplitude of the pulses to zero or by setting theirpulse length to zero. Minimal energy can be zero energy or an energylevel that is small enough such that the pulses do not interfere withthe ability of the ultrasound imaging machine to accurately detect thephotoacoustic signals.

In some embodiments, the ultrasound imaging machine behaves as if ittransmitting ultrasound into the body and detecting the correspondingecho signals when in fact the received echo signals are being generatedin response to the one or more laser pulses. The laser pulse firings aresynchronized with the transmission of the minimal energy ultrasoundbeams. In other embodiments, the transmit electronics of the ultrasoundimaging machine are disabled or suspended at times when the laser pulsesare transmitted.

In some embodiments, the processor 117 is programmed to operate thetransmit electronics 114 in a mode where photoacoustic lines arealternatively received with B-mode lines but no ultrasound pulses aretransmitted. Many ultrasound imaging machines have the ability toalternate firings for B-mode images with another imaging mode. Forexample, many ultrasound machines can operate in Color Power Doppler(CPD) mode where lines to produce a CPD image frame are interleaved withB-modes lines. However, because the photoacoustic signals are receivedwithout a transmission from the ultrasound transducer, the processorchanges the transmit amplitude and/or the duration of the transmitpulses for the PA lines to zero so that minimal or no ultrasound energyis delivered by the ultrasound transducer 120 at the time when the laserpulse are being delivered. In addition, the processor is programmed toproduce a trigger signal each time the system fires these zero energypulses. The trigger pulses are transmitted via a wired or wirelesscommunication link to the external laser light source 140 where thepulses are used to trigger the production of corresponding laser pulses.

The receive electronics 116 are instructed to function once a lightpulse is fired by the laser source through a T/R switch or other means.In some embodiments, for example, the T/R switch 156 is open duringtransmit to protect the receive electronics 116 from high voltages orother electrical anomalies. Once the transmit pulse is fired, thereceive electronics 116 begin to process the echo signals received bythe imaging transducer 120. A system timer starts to store the receivedsignals at a time that equates to a desired distance in the region ofinterest assuming a constant sound speed in tissue of 1540 m/s. In oneembodiment, time zero is equated with echo signals at the skin line andat a maximum time, echo signals are equated to the deepest depth in theregion of interest (e.g. in a range of 1 to 30 cm).

For photoacoustic imaging, the processor 117 sends or causes otherelectronics to send, a trigger signal to the laser light source 140 eachtime an ultrasound pulse is produced (though the transmitter isprogrammed to transmit pulses with minimal or zero energy) for each ormulti-receive lines and each time a laser pulse is requested. The laserlight source 140 fires the laser once the trigger signal is received. Asshown in FIG. 2C, an appropriate trigger delay Δt is used to compensatefor a delay between the time when the trigger signal is transmitted tothe external laser light source 140 and the time at which the laserpulse is actually fired. This delay could be determined based on atheoretical analysis value or an empirical value selected to align thephotoacoustic image frame with the normal background gray-scale tissueimage frame.

The received echo signals for each imaging frame go through beamforming,signal and image processing chains for each line to form an image frame.Multiple receive lines can be formed by each laser firing depending onthe system architecture. Most conventional ultrasound imaging machinescan form 1-4 receive lines from each transmit beam. In one embodiment, areceive line is created for each laser firing. In this mode, the laseris fired multiple times (e.g. ranging from 32 to 256 times) to create aframe of photoacoustic imaging data.

The modifications for the NTV mode include at least a receive delaycalculation based on one-way propagation instead of two-way propagation,received sample registration of wave propagation times, scan conversionmodifications if a different number of lines are used for thephotoacoustic frame from the normal B-mode frame, color map forphotoacoustic image, photoacoustic/tissue decision,blending/combination, CINE for PA+B-mode, clip storage, etc.

Care must be taken to align the PA axial samples in space since it is aone-way data. If the same RF sampling frequency is used (most machinesuse 40 MHz or so) for photoacoustic data as B-mode data, the spacing ofthe photoacoustic axial samples is twice the distance between thesamples as that of the B-mode samples. Either up-sampling or half thedecimation rate can be used to have the same number of RF samples of thePA frame as that of the B-mode frame. The number of lines per framecould be the same or different for the photoacoustic mode. The morelines the better the shape of the tip image but the frame rate slows.

FIG. 2B shows two different image processing paths used to process theB-mode line data and the photoacoustic line data (also referred to asNTV data) in accordance with an embodiment of the disclosed technology.Ultrasound and photoacoustic echo data are received through thetransmit/receive switch 156 and converted to a digital format by ananalog to digital converter 200. Digitized ultrasound and photoacousticecho data may or may not be stored in an RF ultrasound memory 202 (inconventional ultrasound, they are not stored). The stored or real timedata are then processed in slightly different imaging paths. Both theB-mode and the photoacoustic data are processed in a front-end imageprocessor 210 (e.g. DSP, GPU or the like) that performs receivebeamforming, adjusts the gain with a digital gain function, performs I/Qdetection, amplitude detection and decimation. Note the receive delayapplied to the photoacoustic samples is different from that of theB-mode. It is calculated or pre-stored table for one-way propagationinstead of two-way.

The B-mode echo data is then subjected to additional processing in amid-processor 220 where the data are processed by spatial compounding,lateral filtering and up-sampling. Speckle reduction is also performedand a gray-scale map is applied to the B-mode line data. In oneembodiment, the processed echo data and the photoacoustic/NTV line dataare stored in a memory until the frames are completed.

Once the data is complete for a B-mode image frame and aphotoacoustic/NTV image frame, the line data for both image frames areprocessed in a processor 240 by applying a persistence algorithm (itcould be different algorithm or setting for each mode) and scanconversion to produce pixel data for a B-mode image and aphotoacoustic/NTV image. Data for the NTV and B-mode image pixels aresubjected to a different color map, where the needle tip pixels aredisplayed in a system or user selected color. Data for each image arethen analyzed on a pixel by pixel basis to make a tissue/NTV decisionthat determines if the pixel data for the B-mode image or the NTV imageshould be displayed in a particular pixel location of a combined image.The tissue/NTV decision is very similar to the tissue/flow decision thatcould be a simple threshold-based algorithm or some more sophisticatedalgorithm based on multiple parameters. More sophisticated blending alsocould be used such as alpha blending to create a transparent image.

Combined image and NTV pixel data are then processed in a back endprocessor 250 that performs such video processing functions as JPEGcompression before being displayed on a video monitor/display 111,stored on a computer readable medium or transmitted to a remote locationviewing or storage.

FIG. 2D shows one possible implementation of a tissue/NTV data decisionin accordance with an embodiment of the disclosed technology. In oneembodiment, echo data in a photoacoustic frame tested to determine ifthe echo intensity is above a threshold that defines a needle tip. Ifso, the data is marked as NTV data, otherwise the echo data is marked ashaving a zero intensity. B-mode image data and NTV image data are thencombined by replacing B-mode data in a combined image with NTV data. Aswill be appreciated, other more sophisticated methods of decision andblending the B-mode and photoacoustic data could also be performed suchas by alpha-blending or the like.

As described above, in some embodiments of the disclosed technology, theultrasound imaging machine 110 is configured to support line-triggeredlaser firing and to use conventional beamforming paths to formphotoacoustic image lines for each frame. In some embodiments, theultrasound machine 110 can be configured to send out a trigger signalfor each photoacoustic line to be acquired alternately with the B-modefirings. In other embodiments, the ultrasound system acquires a B-modeimage frame and then acquires a NW image frame using one or more laserfirings.

There can be advantages to using a line-based method versus aframe-based method depending on the system architecture. One advantageof the line-triggered method is that it utilizes an existing systemdesign and signal processing chains. No hardware changes are needed toimplement photoacoustic imaging even for older ultrasound systems orportable systems that lack CPU/GPU/FPGA processing power or a largememory to store radio frequency element data or a fast bus to transferthe data to different parts of the computational devices. Moreover, aline-triggered photoacoustic imaging system may not require a largememory to store RF element data nor a fast CPU/GPU/ASIC/FPGA to performsoftware beamforming. The software-based beamforming using FFT algorithmapplies only to linear transducers. For curved and phased transducers,the traditional delay-and-sum method has to be used. But this algorithmis slow to be done by software. The line-triggered method alleviatesthese kinds of issues. In some embodiments, for example, the disclosedtechnology is implemented on a conventional lower-cost and/or portableultrasound imaging machine. In some embodiments, the line-triggeredmethod can be performed using hardware-based receive beamforming thatincludes a delay-and-sum method that can be applied to different typesof transducers (e.g., linear, curved and/or phased).

External Laser Light Source

FIG. 3 is functional block diagram of an external laser light source350. In the embodiment shown, the laser light source includes threeprinted circuit boards: a control board 352, a laser driver board 254and a power board 356. The power board 356 is configured to receive asupply voltage from a connected ultrasound imaging machine and providethe appropriate power levels to run the circuitry in the laser lightsource. The control circuit board 352 includes an FPGA or a processorwith an external or built-in memory that is configured to communicatewith the attached ultrasound imaging machine and to receive the triggersignals which cause the laser to fire. The laser driver board 354 isconfigured to receive control signals from the control board 352 andsupply the appropriate driving signals to the laser source to producelaser pulses when requested. In one embodiment, the external laser lightsource 350 is configured to communicate with the ultrasound imagingmachine using the I2C communication protocol via a cable that isconnected to an EKG port on the ultrasound imaging machine. Other wiredor wireless communication protocols and ports could be used.

A receptacle optical assembly 360 is configured to receive astandardized style of fiber connector having one or more optical fibersthat extend to the distal tip of an interventional instrument (e.g. aneedle). The receptacle assembly includes a laser source such as a laserdiode 362 and an optical coupler and lens that direct light from a laserdiode into the optical fibers. In addition, one embodiment of theoptical assembly includes a micro-switch 366 that closes when theoptical fibers are connected to the optical assembly 360. In oneembodiment, the optical assembly 360 also includes a mechanical orelectrical optical shutter 368 that is placed between the laser diodeand the optical fibers and that is physically moved or made transparentwhen the optical fibers are connected. An LED printed circuit board 358supports a number of LEDs that are illuminated depending on theoperating condition of the laser light source as will be described.

Further details of the external laser light source can be found incommonly owned U.S. patent application Ser. No. ______ (attorney docketnumber 28798-8153 titled “LASER LIGHT SOURCE FOR INSTRUMENT TIPVISUALIZATION”, which is filed concurrently herewith and which is hereinincorporated by reference in its entirety).

As indicated above, the laser light source 350 is configured to producelaser pulses that are transmitted via the one or more optical fibers 142to the tip 135 of the interventional instrument 130. Upon connecting thelaser light source to the ultrasound imaging machine, the controlprinted circuit board 352 is programmed to generate or recall a deviceID from a memory and transmit it to the connected ultrasound imagingsystem. The device ID informs the ultrasound imaging machine that theunit is cable of producing laser pulses for needle tip visualization.Upon receipt of the device ID, the ultrasound imaging machine activatesthe photoacoustic imaging mode. Once a user has inserted an opticalfiber into the laser light source, the control printed circuit boardsends a signal to the ultrasound machine that informs it that it isready to produce laser pulses. As indicated above, the micro-switch 366in the optical assembly is positioned so that it changes state when theuser inserts the optical fibers into the optical assembly. If theoptical fibers are purposely or accidently removed from the laser lightsource 350, the micro-switch 366 changes state again and a signal issent to the connected ultrasound system to suspend or halt thephotoacoustic/NTV imaging process.

In some embodiments, the micro-switch 366 can be replaced with anothertype of sensor (e.g., a relay, a Hall effect sensor etc.) in the opticalassembly that detects proper connection of the one or more fibers 142.In certain embodiments, the software that initiates the photoacousticimaging mode is configured to start a procedure upon detection of anoptical fiber(s) to allow the system start the laser firing, without theoperator having to use a sterile hand or to rely on a helper to initiatethe procedure via the user interface.

Once the fibers are inserted into the laser light source, the controlprinted circuit board 352 monitors the communication connection to theultrasound imaging machine for the trigger signals that indicate when alaser pulse should be fired. Upon receipt of a trigger signal, thecontrol board 352 causes the laser diode to fire one or more pulses intothe connected optical fibers. As discussed above with reference to FIGS.1A-1C and 2, the laser signals are absorbed by the light absorbingmedium 148, which generates corresponding photoacoustic signals that aredetected by the ultrasound imaging machine 110. The ultrasound imagingmachine 110 uses the detected photoacoustic signals to form anultrasound image of the tip 135 of the interventional instrumentrelative to surrounding tissue

FIG. 4 is a screenshot of an exemplary user interface 411 produced bythe ultrasound imaging machine 110 (FIGS. 1A and 1C). The user interface411 includes an ultrasound image 412 showing both tissue 413 in theregion of interest and a colorized tip of the interventional instrument414 (e.g., the tip 135 of FIGS. 1A-1C). A plurality of user inputcontrols 468 can be configured to receive touch inputs from an operatorto control various functions of the ultrasound imaging machine 110.Indicators 465 (identified individually as a first indicator 465 a, asecond indicator 465 b and a third indicator 465 c) provide visual cuesthat the laser light source is connected and powered (465 a), has anoptical fiber connected (465 c) and that the laser is firing (465 c). Insome embodiments, the ultrasound system 110 can be configured to provideone or more tactile, light, color change of the graphic 414 or/and audiofeedback could confirm proper operation. In some embodiments, the colorof the user interface on the ultrasound imaging machine provides cuesabout the NTV accessory connected. For example, text in gray shows thatthe laser light source is connected and is executing the proper softwareversion and the applicable transducer is present and the proper examtype has been selected. White lettering on the user interface indicatesthat the needle is present but the laser is not firing or the system isin freeze mode. Yellow lettering on the user interface indicates thatthe laser is firing. These color text indicators could be saved togetherwith the image for record.

In some embodiments, the ultrasound imaging machine 110 provides gaincontrol for the colorized tip ultrasound image to provide the user ameans to trade off sensitivity with noise. For example, for a deepertarget, the user might want to use higher gain to boost the sensitivitybut tolerate more noise and artifacts. In one embodiment, the thirdindicator 464 c is an indicator for the NTV gain control while 464 a and464 b are for normal tissue and overall gain control.

FIG. 5 is an exemplary flow diagram of a process 500 of forming anultrasound image having a colorized tip ultrasound image indicative of atip of an interventional instrument (e.g., the tip 135 of theinterventional instrument 130). In some embodiments, the process 500 isimplemented with instructions stored in memory (e.g., memory 121 of FIG.1C) and configured to be executed by a processor (e.g., the processor117 of FIG. 1C) of an ultrasound imaging system (e.g., the ultrasoundsystem of FIGS. 1A and 1C). In other embodiments, the process 500 cancomprises instructions stored in hardware and/or software on a suitabledevice or computer (e.g., a medical imaging device) or configuration ofan FPGA.

At step 510, the process 500 transmits a number N (e.g., 128, 256, 512)of ultrasound beams (e.g., the pulses 118 of FIG. 1C) from an ultrasoundtransducer (e.g., the transducer 120 of FIGS. 1A and 1C) toward a regionof interest in a subject.

At step 520, the process 500 transmits a predetermined number ofcorresponding laser light pulses from an external laser light source(e.g., the external laser light source 140 of FIG. 1A). As describedabove in reference to FIGS. 1A-1C, individual laser light pulses can betransmitted via one or more optical fibers that terminate at the lightabsorbing medium (e.g., a black epoxy) at a tip of an interventionalinstrument. The light absorbing medium absorbs the laser light pulsesand generates a corresponding photoacoustic signal surrounding the tipof the interventional instrument. In some embodiments, the process 500operates in a line-based imaging mode and transmits a laser light pulsethat is interleaved with each of the number N (e.g., 32 to 512 is atypical range of number of lines per frame) of ultrasound beamstransmitted at step 510. In other embodiments, the process 500 operatesin a frame-based imaging mode and transmits a number of laser lightpulses that are not interleaved with the number N ultrasound beams atstep 510. In photoacoustic mode imaging, the transmitter is shut downand there is minimum or no ultrasound energy output from the transducer.The entire photoacoustic image frame could be interleaved with normaltissue imaging frame: B+PA+B+PA . . . , therefore, the transmitter isshut down every other frame. In certain embodiments, however, a block ofphotoacoustic lines could be interleaved with a block of normal tissueimaging lines. In some embodiments, the process 500 operates in asubframe-based imaging mode and transmits one or more laser light pulses(e.g., 4, 8, 16, 32) and then transmits and receives of a number of Nultrasound beams at step 510 for normal tissue imaging.

At step 530, the process 500 acquires ultrasound data corresponding toultrasound echoes (e.g. detected ultrasound echoes 119 of FIG. 1C)received from the region of interest. The process further acquiresphotoacoustic data corresponding to the photoacoustic signals generatedfrom the tip of the interventional instrument at step 520.

At step 540, the process 500 constructs a combined image using theacquired ultrasound echoes and the photoacoustic data acquired at step530. A photoacoustic/B-mode decision has to be made to each pixel beforecombining them. A simple threshold or a complex fuzzy logic algorithmcould be used.

At step 550, the process 500 combines and/or blends the data from thetwo imaging modes at step 540 to form a combined image.

At step 560, the process 500 outputs the combined image to a display(e.g., the display 111 of FIG. 1C) as shown, for example, in FIG. 4. Thecombined image includes image data corresponding to background tissue inthe region of interest and colorized image data corresponding to the tipof the interventional instrument. In some embodiments, the backgroundtissue is shown in gray while the image data corresponding to the tip ofthe interventional instrument is shown in an operator-desired color(e.g., red, yellow, blue, orange). In some embodiments, yellow is thepreferred color for the pixels representing the instrument tip becauseyellow is not generally associated with other ultrasound imaging modesand is easily seen by users—even those with red and green color impairedvision.

From the foregoing, it will be appreciated that specific embodiments ofthe invention have been described herein for purposes of illustration,but that various modifications may be made without deviating from thescope of the invention. Accordingly, the invention is not limited exceptas by the appended claims.

1-17. (canceled)
 18. A system, comprising: a laser light sourceconfigured to produce laser light pulses; one or more optical fibershaving a first end and a second end, the first end configured to beoptically coupled to the laser light source, the second end configuredto be attached to an interventional instrument and to producephotoacoustic signals in response to receiving the laser light pulses; atransducer configured to transmit ultrasound signals toward a region ina subject and to receive: ultrasound echoes from the subject based onthe ultrasound signals; and the photoacoustic signals; and an ultrasoundmachine configured to: generate, based on the ultrasound echoes and thephotoacoustic signals, an image that depicts an anatomical feature inthe region and a representation of the interventional instrument; andsynchronize the laser light source with the transducer including toinstruct the transducer to generate the ultrasound signals with areduced energy when the laser light source produces the light pulses.19. The system of claim 18, wherein the ultrasound machine is configuredto synchronize the laser light source with the transducer including togenerate trigger signals and transmit the trigger signals to the laserlight source, and the laser light source is configured to produce thelaser light pulses upon receipt of the trigger signals.
 20. The systemof claim 18, wherein the second end of the one or more optical fibersincludes a light absorbing medium configured to produce thephotoacoustic signals.
 21. The system of claim 18, wherein thetransducer is configured to generate the ultrasound signals with thereduced energy by reducing an amplitude of the ultrasound signals. 22.The system of claim 18, wherein the transducer is configured to generatethe ultrasound signals with the reduced energy by reducing a duration ofthe ultrasound signals.
 23. The system of claim 18, wherein theultrasound machine is configured to receive a signal from the laserlight source indicating that one or more optical fibers have beeninserted into the laser light source and to begin operating in a needlevisualization mode responsive to said receive the signal from the laserlight source.
 24. The system of claim 18, wherein the ultrasound signalsinclude ultrasound beams, and said synchronize the laser light sourcewith the transducer includes to interleave the laser light pulses withthe ultrasound beams, wherein the ultrasound machine is configured toreceive the ultrasound echoes interleaved with the photoacousticsignals.
 25. The system of claim 18, wherein the ultrasound machine isconfigured to colorize the anatomical feature and the representation ofthe interventional instrument separately for the image.
 26. The systemof claim 18, wherein the interventional instrument includes a needle andthe second end is positioned adjacent a tip of the needle.
 27. Thesystem of claim 18, wherein the ultrasound machine is configured togenerate a trigger delay to compensate for a time delay between a firsttime at which a trigger signal is transmitted to the laser light sourceand a second time at which a laser light pulse is generated by the laserlight source in response to the trigger signal to align a photoacousticimage frame associated with the photoacoustic signals with an ultrasoundimage frame associated with the ultrasound echoes.
 28. The system ofclaim 18, wherein the ultrasound machine is configured to determine thatan intensity of photoacoustic data associated with the photoacousticsignals is greater than a predetermined threshold to mark thephotoacoustic data as corresponding to the interventional instrument.29. An ultrasound machine, comprising: a transducer configured totransmit ultrasound signals toward a region in a subject and to receiveultrasound echoes from the subject based on the ultrasound signals; anda processor configured to: combine echo data based on the ultrasoundechoes and photoacoustic data based on photoacoustic signals producedresponsive to laser light pulses from a laser light source to create animage that shows an anatomical feature in the region and at least aportion of an interventional instrument; generate trigger signals toinstruct the laser light source to generate the laser light pulses; andgenerate control signals to instruct the transducer to reduce an energyof the ultrasound signals transmitted by the transducer to a minimalenergy to synchronize the laser light pulses with the ultrasound signalshaving the minimal energy transmitted by the transducer.
 30. Theultrasound machine of claim 29, wherein the transducer is configured toreduce the energy of the ultrasound signals to the minimal energy byreducing an amplitude or a duration of the ultrasound signals inresponse to the control signals.
 31. The ultrasound machine of claim 29,wherein the processor is configured to begin operating in an instrumentvisualization mode upon receipt of a signal from the laser light sourcethat indicates that one or more optical fibers have been connected tothe laser light source.
 32. The ultrasound machine of claim 29, whereinthe processor is configured to stop operating in an instrumentvisualization mode upon receipt of a signal from the laser light sourcethat indicates that one or more optical fibers have been disconnectedfrom the laser light source.
 33. The ultrasound machine of claim 29,wherein the processor is configured to interleave the laser light pulseswith the ultrasound signals on an image frame basis by interleaving anamount of the laser light pulses corresponding to a photoacoustic imageframe with an amount of the ultrasound signals corresponding to anultrasound image frame.
 34. A method of operating an ultrasound machine,the method comprising: transmitting, with a transducer of the ultrasoundmachine, ultrasound signals toward a region of a subject; acquiringultrasound echo data based on ultrasound echoes that are received by thetransducer in response to the ultrasound signals transmitted; generatingtrigger signals to control a laser light source to produce laser lightpulses, wherein the laser light pulses are transmitted from a first endof one or more optical fibers that is coupled to the laser light sourceto a second end of the one or more optical fibers that is coupled to aninterventional instrument to produce photoacoustic signals in responseto the laser light pulses; generating control signals to synchronize thetransducer and the laser light source, the control signals indicating tosuspend transmission of the ultrasound signals or reduce an energy ofthe ultrasound signals transmitted by the transducer to a minimal energywhen the laser light source produces the laser light pulses; acquiringphotoacoustic data from the photoacoustic signals; and combining theultrasound echo data and the photoacoustic data to produce an image thatshows an anatomical feature in the region and at least a portion of theinterventional instrument.
 35. The method of claim 34, furthercomprising: transmitting the trigger signals to the laser light sourceover a wired or wireless communication link.
 36. The method of claim 34,further comprising: receiving a signal from the laser light sourceindicating that the one or more optical fibers have been removed fromthe laser light source; and automatically halting, upon receipt of thesignal from the laser light source, a photoacoustic imaging process thatincludes said generating the trigger signals.
 37. The method of claim34, further comprising: determining a location of a tip of theinterventional instrument based on the photoacoustic signals and notbased on the ultrasound echoes.